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[57] ABSTRACT 

A method and apparatus for generating keyboard commands 
in response to voice commands is described. An association 
between command strings and a set of keyboard combina- 
tions is established. Each command string is associated with 
a keyboard combination. Command strings are generated by 
a sound engine in response to voice commands. A keyboard 
combination is selected based on the association between 
keyboard combinations and command strings. The keyboard 
combination is sent to a computer system as mimicked 
signals from a keyboard. The keyboard combinations are 
mimicked as signals from the keyboard by sending the 
keyboard combination to a keyboard controller and then 
sending a message to the keyboard controller to send the key 
combination as signals generated from the keyboard. 
Alternatively, a keyboard message may be sent to the 
operating system running on the computer system, causing 
the computer system to respond to the keyboard combina- 
tion as signals generated from a keyboard. The sound engine 
receives a voice command while the sound engine is gen- 
erating sound. The command string associated with the 
command string is found a ad then transmitted. The com- 
mand strings are associated with keyboard combinations that 
control the play of a computer game. 

10 Claims, 7 Drawing Sheets 
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MIMICKING VOICE COMMANDS AS The need to issue the latter command does not occur as often 

KEYBOARD SIGNALS as the need to issue the commands in the select set. However, 

because the need may occasionally occur during a dogfight, 

HELD OF THE INVENTION the user's ability to play effectively is impaired. 

The present invention relates to voice recognition by 5 One approach to overcome the previously discussed prob- 

computer systems, and in particular, to generating keyboard lem ^ lne integrated voice recognition approach. In the 

commands from voice commands. integrated voice recognition approach, the computer game 

software is configured to accept commands from a particular 
BACKGROUND OF THE INVENTION sound engine. As used herein, the sound engine refers to the 
Computer games are a very popular form of entertain- 10 hardware and software components on a computer system 
ment. Action oriented computer games, such as a flight speafically responsible for receiving and mterpreting sound 
simulators for war planes, are becoming more realistic and m P ut and *°™* 0Ut P u j- Fo ; exam P lc > a computer 
more sophisticated. Usually such games are played with s ^ m md ? include a 50X111(1 card J ™ d sound card *>ti»m. 
input devices designed for computer game playing ("game 5 The sound engine receives sound from an externa device, 
input devices"), such as a joystick control and game pads. 15 such 35 a microphone, and transmits data typically repre- 
_ , senting spoken words to an application running on the 
To control the play of the game, the computer game * le f lem . ^ be havior „ f Ae application is con- 
software is configured to receive commands from the game ^ ^ Qa ^ ^ {mm ^ ^ 
player (user). Typically, the game software accepts as of of in addjtion , 0 ^ ^ &om , ^ 
commands from the user, signals from the keyboard, or 20 

signals from a game input device. Many game input devices m , , „, 

provide buttons and other control mechanisms, such as a pan J The "^grated voice recognition approach offers several 

button, which are manipulated by the user to send a signal advantages over the use of a keyboard during game play, 

to the computer that the computer game software recognizes ^ commands not in the select set may be *sued vocally 

as a command. For example, a trigger button on a joystick 25 ™ ihom the _ u f r havm S 1° m0 * e * hmd ° S l £ g3m ! mpUt 

attached to a joystick control may be depressed by the user device , t0 < he keyboard. Second, the user is alleviated from 

toafirethe weaponsof fighter jet. One button on the joystick the , task of memorizing the various combinations of key 

control may be depressed to select a weapon, a second ****** Deeded 10 issue the man y commands typically 

button may arm the weapon, a third button may disarm the needed to P la y a computer game. For example, rather than 

weapon switch to a top view by pressing Shift -T, the user can simply 

^ say "top view" 

Typically the number of commands recognized by a \ ,f , _ t t 
computer game exceeds the number of controls that can be disadvantage of the integrated voice recognition 
implemented on a game input device. Thus computer game a PP roach 15 computer game software must be specifically 
software is typically configured to recognize a select set of Programmed to interface with and receive input from sound 
commands as signals from game input devices. The remain- 35 en ? mCS ' ^ CaUSC ° f the CXtra f 0051 ° f develo P in S 
der of the commands may be issued as signals from the software that rec f ives m P ut . from 50110(1 en S mes ' man y 
keyboard, The select set of commands typically includes computer games do not offer integrated voice recognition, 
those commands that must be quickly issued in the various Clearly, it is desirable to provide a mechanism that 
situations presented by the computer game. For example, provides the ability to vocally issue commands when play- 
one common situation presented by a computer game is a 40 in g computer games that are not configured to interface with 
simulated dogfight. During a simulated dogfight, a user may a soun d engine. 

need to quickly issue many commands, such as commands CI r w X * A1W ™ TXT , /T?xrr ~~ XT 

, t . , , * , SUMMARY OF THE UvvtNTION 
to control the airplane s thrust, airbrakes, rudder, radar, and 

weapons. When all the needed commands are part of the A method and apparatus for generating keyboard corn- 
select set, the commands can be quickly issued through the 45 mands in response voice commands is provided. According 
game input device. The user does not have to move a hand to one aspect of the invention, an association between 
to the keyboard to issue any command needed in the current command strings and a set of keyboard combinations is 
situation facing the user. In light of fast pace of dogfight, the established. Each command string is associated with a 
player can play much more effectively if all the needed keyboard combination. 

commands may be quickly issued through the game input 50 Command strings are generated by a sound engine in 

device. response to voice commands. A keyboard combination is 

Because the number of commands that can be part of the selected based on the association between keyboard combi- 

select set is limited, often in particular situations presented nations and command strings. The keyboard combination is 

by computer games, there are commands that users need to sent to a computer system as mimicked signals from a 

issue which are not part of the select set. Thus, in situations 55 keyboard. The keyboard combinations are mimicked as 

where a command is not part of select set and the need for signals from the keyboard by sending the keyboard combi- 

the user to issue the command arises, the user must take one nation to a keyboard controller, and then causing the key- 

or both hands off the game input device and move them to board controller to send the keyboard combination as signals 

the keyboard to issue the command through the keyboard. In generated from the keyboard. Alternatively, a keyboard 

these situations, the user's ability to play effectively is 60 message may be sent to the operating system running on the 

impaired. For example, assume that commands that are most computer system, causing the computer system to respond to 

often needed in a simulated dogfight, such as commands that the keyboard combination as signals generated from a 

control the movement of the fighter jet and fire the air-to-air keyboard. 

missiles and machine guns, are part of the select set of According to another aspect of the invention, the sound 

commands. However, the command that pans the view 65 engine receives a voice command while the sound engine is 

presented to the user on the computer screen to a perspective generating sound. The command string associated with the 

looking up from the jet fighter is not part of the select set. command string is found and then transmitted. The com- 
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mand strings are associated with keyboard combinations that 
control the play of a computer game. 

According to another aspect of the invention, a command 
string, keyboard combination, and data identifying a type of 
keyboard manipulation is received from a user. A mapping 
of the command string to the keyboard combination and type 
of keyboard manipulation is stored in the computer system. 
If the type of keyboard manipulation is the "hold" type, a 
duration is received from the user. 

According to another aspect of the invention, if the 
keyboard combination is mapped to a hit type of keyboard 
manipulation, then the keyboard combination is sent to the 
computer system mimicked as a key combination that is 
depressed and immediately released. A repeat count may be 
received from the user. The sending of the keyboard com- 
bination to mimic a keyboard signal that is depressed and 
immediately released is repeated for the number of times 
corresponding to the repeat count. 

According to another aspect of the invention, if the 
keyboard combination is mapped to a hold type of keyboard 
manipulation with no specified duration, then the keyboard 
combination is sent to the computer system mimicked as a 
key combination that is depressed indefinitely. A subsequent 
command string from the sound engine is received, the 
subsequent command string being associated with a release 
type of keyboard manipulation. In response to receiving the 
subsequent command string, another keyboard combination 
is sent to mimic a signal generated from the keyboard that 
the keyboard combination that is depressed indefinitely is no 
longer depressed. 

According to another aspect of the invention, if the 
keyboard combination is mapped to a hold type of keyboard 
manipulation and a duration greater than zero, then the 
keyboard combination is sent to the computer system to 
mimic a key combination that is depressed for a period of 
time and then released. The sending of the keyboard com- 
bination to mimic a keyboard signal that is depressed, held 
for a period of time, and then released is repeated for the 
number of times corresponding to the repeat count. 

According to another aspect of the invention, an associa- 
tion between a command string and a set of keyboard 
combinations is established. An association between the 
command siring and a repeat count is also established. The 
keyboard combination is sent to the computer system, for the 
number of times indicated by the repeat count, as mimicked 
signals from the keyboard. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention is illustrated by way of example, 
and not by way of limitation, in the figures of the accom- 
panying drawings and in which like reference numerals refer 
to similar elements and in which: 

FIG. 1A is a block diagram of a computer system on 
which the present invention may be implemented; 

FIG. IB is block diagram showing particular components 
of a computer system on which the present invention may be 
implemented; 

FIG. 2 is a block diagram showing objects used to 
represent command sets; 

FIG. 3 is table showing attribute values of exemplary key 
objects and the command string attribute values of the 
command objects which contain the key objects; 

FIG. 4 is a flowchart showing steps performed to collect 
data for creating command sets; 

FIG. 5 is a flowchart showing the steps performed in 
response to receiving voice commands from a sound engine; 
and 
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FIG. 6 is a flowchart showing the steps performed to 
mimic key combinations as signals from a keyboard. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENT 

A method and apparatus for generating keyboard com- 
mands from spoken commands is described. In the follow- 
ing description, for the purposes of explanation, numerous 
specific details are set forth in order to provide a thorough 
1 understanding of the present invention. It will be apparent, 
however, to one skilled in the art that the present invention 
may be practiced without these specific details. In other 
instances, well-known structures and devices are shown in 
block diagram form in order to avoid unnecessarily obscur- 
15 ing the present invention. 

HARDWARE OVERVIEW 

FIG. 1 is a block diagram that illustrates a computer 
system 100 upon which an embodiment of the invention 

20 may be implemented. Computer system 100 includes a bus 
102 or other communication mechanism for communicating 
information, and a processor 104 coupled with bus 102 for 
processing information. Computer system 100 also includes 
a main memory 106, such as a random access memory 

25 (RAM or other dynamic storage device, coupled to bus 102 
for storing information and instructions to be executed by 
processor 104. Main memory 106 also may be used for 
storing temporary variables or other intermediate informa- 
tion during execution of instructions to be executed by 

30 processor 104. Computer system 100 further includes a read 
only memory (ROM) 108 or other static storage device 
coupled to bus 102 for storing static information and instruc- 
tions for processor 104. A storage device 110, such as a 
magnetic disk or optical disk, is provided and coupled to bus 

35 102 for storing information and instructions. 

Computer system 100 may be coupled via bus 102 to a 
display 112, such as a cathode ray tube (CRT), for displaying 
information to a computer user. An input device 114, includ- 

^ ing alphanumeric and other keys, is coupled to bus 102 for 
communicating information and command selections to 
processor 104. Another type of user input device is cursor 
control 116, such as a mouse, a trackball, or cursor direction 
keys for communicating direction information and com- 

45 mand selections to processor 104 and for controlling cursor 
movement on display 112. This input device typically has 
two degrees of freedom in two axes, a first axis (e.g., x) and 
a second axis (e.g., y), that allows the device to specify 
positions in a plane. 

50 The invention is related to the use of computer system 100 
for generating keyboard commands from spoken commands. 
According to one embodiment of the invention, the genera- 
tion of keyboard commands from spoken commands is 
provided by computer system 100 in response to processor 

55 104 executing one or more sequences of one or more 
instructions contained in main memory 106. Such instruc- 
tions may be read into main memory 106 from another 
computer-readable medium, such as storage device 110. 
Execution of the sequences of instructions contained in main 

60 memory 106 causes processor 104 to perform the process 
steps described herein. In alternative embodiments, hard- 
wired circuitry may be used in place of or in combination 
with software instructions to implement the invention. 
Thus, embodiments of the invention are not limited to any 

65 specific combination of hardware circuitry and software. 
The term "computer-readable medium" as used herein 
refers to any medium that participates in providing instruc- 
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tions to processor 104 for execution. Such a medium may work link 120 and communication interface 118. In the 

take many forms, including but not limited to, non-volatile Internet example, a server 130 might transmit a requested 

media, volatile media, and transmission media. Non-volatile code for an application program through Internet 128, ISP 

media includes, for example, optical or magnetic disks, such 126, local network 122 and communication interface 118. In 

as storage device 110. Volatile media includes dynamic 5 accordance with the invention, one such downloaded appli- 

memory, such as main memory 106. Transmission media cation prov ides for generation of keyboard commands from 

includes coaxial cables, copper wire and fiber optics, includ- spoken commands as described herein, 

ing the wires that comprise bus 102. Transmission media can ^ feceived co(Je ^ M 

also take the form of acoustic or light waves, such as those . . . , , J , . , J K 

. * , • j * A j ^Ir- « J . . it is received, and/or stored in storage device 110, or other 

generated during radio- wave and infra-red data commum- „ , c . 

cations 1 Q°n-volatile storage for later execution. In this manner, 

~ c r . ... j -ijr computer system 100 may obtain application code in the 

Common forms of computer-readable media include, for f orm 0 f a carrier wave 

example, a floppy disk, a flexible disk, hard disk, magnetic „ _ . 1nri/ , * . , ,• 

tape, or any other magnetic medium, a CD-ROM, any other * c . fc ™e t0 FIG * 1B > FIG * 1B * * block showi u g 

optical medium, punchcards, papertape, any other physical „ additional components upon which an embodiment of the 

medium with patterns of holes, a RAM, a PROM, and 15 invention operates. Keyboard controller 158 is a standard 

EPROM, a FLASH-EPROM, any other memory chip or keyboard controller which may be connected to a keyboard, 

cartridge, a carrier wave as described hereinafter, or any exam P le of Wn** controller is the Intel 8042. 

other medium from which a computer can read. 0ne or more keyboard ports logically connect keyboard 

Various forms of computer readable media may be , n c ° ntroller 158 to bus ™* In this context, the term "port- 
involved in carrying one or more sequences of one or more ^ rs 10 [ he , logical connection between computer system 
instructions to processor 104 for execution For example, the 100 a ° d the ke yk°ard controller 158. Each keyboard port is 
instructions may initially be carried on a magnetic disk of a identified by a number that is used to select and address the 
remote computer. The remote computer can load the instruc- P° rl ""erface. When a key on the keyboard is depressed or 
tions into its dynamic memory and send the instructions over „ released, the keyboard controller 158 receives signals from 
a telephone line using a modem. A modem local to computer the keyboard mdicating which key has been depressed or 
system 100 can receive the data on the telephone line and ^leased. The keyboard controller 158 then uses the key- 
use an infra-red transmitter to convert the data to an infra-red board P° rts 10 communicate the depressed or released key to 
signal. An infra-red detector coupled to bus 102 can receive computer system 100. 

the data carried in the infra-red signal and place the data on 30 Through use of the keyboard ports, the keyboard control- 
bus 102. Bus 102 carries the data to main memory 106, from ler 158 * 31:50 configured to receive signals from computer 
which processor 104 retrieves and executes the instructions. system 100 that to the keyboard controller 158 indicate that 
The instructions received by main memory 106 may option- a P artic ular key on the keyboard has been depressed or 
ally be stored on storage device 110 either before or after released. As shall be discussed in greater detail below, 
execution by processor 104. computer system 100 sends such signals to the keyboard 
Computer system 100 also includes a communication 35 CODtroller 158 to cause ^e keyboard controller 158 to 
interface 118 coupled to bus 102. Communication interface communicate to computer system 100 that the particular key 
118 provides a two-way data communication coupling to a has been de P ressed or released, 
network link 120 that is connected to a local network 122. SOUND ENGINE 
For example, communication interface 118 may be an into- 40 Sound engine 150 includes sound card 152 and voice 
grated services digital network (ISDN) card or a modem to iticm sahwm 154 . A microphone 153 maBjBCIed to 
provide a data communication connection to a correspond- ^ ^ cajd 152 ^ voice ition software 154 and 
ing type of telephone line. As another examp e commum- ^und card 152 t ther afe incoming Frances from 
cationmterfaceU8maybeabcalareanetwork(l^card , user tQ aQ active vocabul t0 determm e whether the 

7™\£ e a^ 45 spoken utterances match any word or phrase in the active 

LAN. Wireless links may also be implemented. In any such vocabulary . If a match fa found the sound en ^ ne 150 

implementation, communication interface 118 sends and {vaQSmits a message t0 computer system 100. The sound 

receives electrical, electromagnetic or optical signals that m£ 150 fc Mc of mQDin {q fau d kx mode 

carry digital data streams representing various types of - „ . , . f 

information duplex, as used herein, refers to the ability of sound 

XT ,',.,,-«.„ j . - 50 engine 150 to generate sound while concurrently receiving 

Network link 120 typically provides data communication ™i™ ° c ft „ A t „ ncm : tt : nn mo «.™«. ,X ,u „™ 

. , *r j r spoken utterances and transmitting messages to the com- 

through one or more networks to other data devices. For 100 jn fe tQ matchin ^ ken ^ 

exampfc network hnk 120 may prov.de a connection ances [0 an active vocabul An^iy, m vtaiM 

through local network 122 to a host computer 124 or to data ^ card ma ^ 0Qe fof from fc 

SfC'r 1 " 1 by an J In, ! niet Serv.ce Provider (ISP) 55 user, the other for generating sound. 
126. ISP 126 in turn provides data communication services 

through the world wide packet data communication network COMMAND SETS AND ENTRY 

now commonly referred to as the "Internet" 128. Local When the user types in a sequence of keystrokes, data 

network 122 and Internet 128 both use electrical, electro- representing the keystrokes is typically delivered from the 

magnetic or optical signals that carry digital data streams. 60 keyboard controller 158 to the software application running 

The signals through the various networks and the signals on on the computer system 100. If the software application 

network link 120 and through communication interface 118, running on the computer system 100 recognizes the data 

which carry the digital data to and from computer system representing the sequence of keystrokes as one or more 

100, are exemplary forms of carrier waves transporting the commands, the software performs one or more correspond- 

information. 65 actions. Thus, by entering a sequence of keystrokes, a 

Computer system 100 can send messages and receive user is able to control the execution of a software applica- 

data, including program code, through the network(s), net- tion. 
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A command set is a set of keyboard commands. A attribute value represents an indefinite period of time (e.g. a 
keyboard command is a sequence of one or more keystrokes zero or nonnumeric value), then the period of time that the 
that may be entered by a user on a keyboard to control the key remains depressed is indefinite (i.e. until a "release" is 
execution of a software application running on the computer issued manner which shall be described), 
system lW. Acco^ 5 ^ third keyboard manipulation type is the "release- 
object 210 is used to represent and manipulate data for a m represents a release of a key 
particular command seL Typically a command set com- A « release « t ^ e of k object ^ 

prises a set of keyboard commands used to control the , „ . . . „. /,„ , . , . 3 

t - c fu. v .* u . created in tandem with a hold type of key object having an 

execution of one software application, such as a computer . , £ . , . A , . */ ■ f . J - . , f .. 

ame indefinite duration. As shall be described in further detail, 

For example, consider an exemplary fighter jet simulator. 30 of key objects are used to logicaUy release a 

By repeatedly depressing and releasing the keys 'Ctrl+R' k ^. that 15 bem 8 <kP«sed ( held ) for an "^definite penod 

five times in sequence, the user causes the fighter jet of * 

simulator to release five sets of flares. Note the "Ctrl" and The repeat count attribute 244 represents the number of 

"R" represent keys on a keyboard. The "+" sign between the times the key combination is repeated. 

Ctrl and "R" is a common form of notation denoting that the 15 The text attribute 248 stores data that represents other 

keys are pressed simultaneously. The sequence of depressing keys that are to be depressed and released after the key 

and releasing 'Ctrl+R* is a keyboard command. combination has been repeated. For example, if the text 

FIG. 2 shows exemplary command set object 210 in attribute of a key object contains "mkb", then a pair of 

greater detail. An object is a record of data combined with key-down and key-up instruction sequences are sent for each 

the procedures and functions that manipulate the record. All 20 character in the "mkb" in the order the character appears in 

objects belong to a class. All objects belonging to the same the text, 
class have the same fields ("attributes") and the same 
methods. The methods are the procedures and functions used 

to manipulate the object. An object is said to be an FIG. 4 is a flow chart showing the steps of a method for 

"instance" of the class to which the object belongs. The 25 creating a command set. In one embodiment of the 

techniques described herein are not limited to object ori- invention, command set entry process 170 (FIG. 1A) pro- 

ented software. For example, object classes correspond to vides a graphical user interface to the user for the purpose of 

abstract data types, objects correspond to instances of collecting data from the user. An example of a graphical user 

abstract data types, and the methods of an object correspond interface is a dialogue box object invoked by software 

to the routines that are used to perform operations on data 30 developed using the Microsoft Foundation Class Library, 

contained in instances of abstract data types. The user enters data by, for example, typing on a keyboard 

Command set object 210 contains data representing a or by manipulating a mouse to point to and click on a 

command set. A command set object has a command set graphical control, such as option button in an option button 

name attribute 212 and a collection of one or more command control, available as part of the Microsoft Foundation Class 

objects 220. Exemplary command object 220 has a com- 35 Library. 

mand string attribute 222 and one or more key objects 228. The steps illustrated in FIG. 4 are performed to collect 
As shall be described in further detail, the command string data used to create command objects, and to collect data 
attribute 222 is used to associate command object 220 with used to create key objects for each of the command objects, 
a voice command. ^ When the desired command objects are created for a corn- 
Command object 220 contains a collection of key objects mand set, the command set, along with command objects 
228, of which key object 230 is an example. A key object and key objects contained by the command set, are stored in 
belongs to the key object class. According to one command file on a static storage device 110 for later retrieval 
embodiment, each key object has the following attributes: in manner which shall be described in further detail, 
key combination 240, keyboard manipulation type 242, 45 To facilitate in the description of the steps shown in FIG. 
repeat count 244, duration 246, and text 248. 4, it shall be assumed that it is desired to create a command 
The key combination attribute 240 stores data that rep- set with a command set name of "Fighter Jet" to be used to 
resents a combination of a keyboard key and key modifiers play a computer game called "Fighter Jet". The command 
that modify the key, such as shift, Ctrl, and Alt keyboard set contains a keyboard command for applying the airbrakes 
keys. 'Ctrl+R* is an example of a key combination. 50 for 3 seconds. A user may issue a command to apply the 
The keyboard manipulation type attribute 242 stores data airbrakes by depressing Ctrl+A for the desired period of time 
which represents the manner in which a user manipulates a me wishes to apply the air brakes, 
key. In an embodiment of the invention, there a three types At step 410, a command set name is received from the 
of keyboard manipulation types. The first is the "hit" type. user. In this example, assume the user types "Fighter Jet" on 
The "hit" type represents a key combination that is 55 the keyboard, and that the command set entry process 170 
depressed and immediately released. An example of "hit" receives the string "Fighter Jet". 

keyboard manipulation type is depressing 'Ctrl+R* to release At step 420, a command string is received. In this 

a flare and immediately releasing it. example, assume the user types "AirBrakes" on the 

The second keyboard manipulation type is the "hold" keyboard, and that the command set entry process 170 

type. The "hold" type represents depressing a key combi- 60 receives the string "AirBrakes". 

nation for a period time before releasing the key combina- At step 430, a keyboard combination is received when the 
lion. Depressing the 'F' key to fire the machine gun in a user depresses the keys corresponding to the desired key- 
fighter jet simulator without releasing the key for two board combination. In this example, the user depresses 
seconds is an example of "hold" keyboard manipulation 'Alt+A', causing the command set entry process 170 to 
type. 65 receive a string corresponding 'Alt+A\ 

The duration attribute 246 stores data that represents the At step 440, data representing a keyboard manipulation 

period of time that a key remains depressed. If the duration type is received. In this example, assume the user uses a 
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mouse to click on a particular option button in option control command object with a command string attribute value of 

presented in the user interface. The particular option button "AirB rakes" is then created and made one of the command 

is designated as the button to click if the keyboard manipu- objects contained by the command set object just created, 

lation type desired is the "bold" type. Command set entry Then a key object is created, having a key combination, 

process 170 receives data indicating that the "hold" type was 5 keyboard manipulation type, repeat count, duration, and text 

selected by the user. attribute values of "Alt+A", "Hit", 1, 3000, and"" respec- 

At step 450, a determination is made of whether the lively. Then the key object is made one of the key objects 
"hold" type was received in at step 440. If the determination contained by the command object just created, 
made is that the "hold" type was not received, then control The collection of key objects contained within a com- 
passes to step 460. If the determination made is that data 10 ma ad set object establishes a mapping between the attributes 
representing the "hold" type was received, then control of each key object in the collection and the command string 
passes to step 460, as is the case in this example. attribute of the command object containing the collection. In 

At step 460, data representing the duration is received. the above example, the key combination, keyboard manipu- 

The data received represents a number of milliseconds. Any lation type, repeat count, duration, and text attributes with 

data received that does not indicate a number considered by 15 the values of "Alt+A", "Hit", 1, 3000, and"" are considered 

the command set entry process 170 to represent a duration mapped to the command string attribute having the value 

of zero. A duration of zero is considered to be a duration for " AirBrakes". 

an indefinite period of time. Control then passes to step 470. At step 480, the command set object, along with the 

In thus example, the user types in the number 3000, and the objects contained by it, are stored in a command file in 

command set entry process 170 receives data indicating that 20 storage device 110. The file is stored under the file name that 

the duration is 3000 milliseconds (3 seconds). corresponds to the command set. Techniques for storing 

At step 470, data representing the repeat count is received. objects in files, and for storing objects under a file name are 

The data received represents a number. Any data received well known to those skilled in the art and shall not be further 

that does not indicate a number is processed by the com- described, 

mand set entry process 170 as if the number "1" had been 25 

received. In this example, the user types in nothing, and ESTABLISHING VOICE COMMAND SET 

command set entry process 170 receives data representing « . 1CA ... 4 c 4 , . , 

tI _ . - . - _ r , & Sound engine 150, like most of those available on the 

that nothing has been entered by the user. Command set „ , . ,. V4 . f . ... . . t e 

nn . n J ..... t_ «,« market, provide utilities for establishing sets of voice utter- 
entry process 170 therefore presumes that the number 1 M / ■ . , . . & , f , 
was entered 30 ' P nrases » words) recognizable by the sound 

. engine. Each voice utterance which is recognized by the 

At step 474, any text entered by the user is received and ^a engine is herein referred t0 as voice commaod Each 

stored in the text attribute 248 of the key object 230. In this voice cornmand fe associated with a command string. For 

example, no text is received. example, the voice utterance represented by the word "fire" 

At step 480, command data is stored in the computer 35 may be associated with the command string "fire", 

system 100 based upon the input received from the user. Voice mtion software 154 ides a ^ imerface 

According to one embodiment of the invention, the com- fof ^ voice commands B ^ md for designating 

mand data is stored in the form of a command set object 210, n • ~L „ nm Z n „A ™» *u„ „ _ t r * M f J* 

... . . t ... a voice command set as the active set or voice commands 

with its associated attributes. In such an embodiment, step M • , , . „ , ^ • /t . 

. 0rt . . , . j-f. presently recognizable by the sound engine (the active 

480 includes creating command set object 210 in memory (if m vocabula »). ^ a ^ uttels a voice (i . e . „ 

not already created) and storing the received information m ut[erance „ recognizable by the sound engine), the 

the appropriate attributes within command set obiect 210. j * . / a 

rif v J sound engme transmits a message to command process 160. 

Specifically, the strmg "Fighter Jet" is stored in the A message transmitte d in response to receiving a voice 

command set name attribute 212 of the command set object command to me command process 160 is herein referred to 

210. If command object 220 does not yet exist, then memory 45 as a voice command message. 

is allocated for command object 220 and the string "Air- . ... t c . . . .... . n _ 

Brakes" is stored in the command string attribute 222 of the ,n ° ne ™t>°diment of the invention, the W,ndows95 

command object 220. If key object 230 does no. exist, then °P eratin S r s y stem ;. whlcb ». S enerall y ava » lab e ^ 

„, . ; . ■ 4 ^, ft ■ .-I * j .1. V. Microsoft Corporation, is running on computer system 100. 

memory key object 230 is allocated and data representing r™. *~ > & . , J , 

*u i u a ~/-.u- ** * a ' *l. i t . The voice command message is transmitted to command 

the keyboard comb mat ion is stored in the key combination <n t_ . L - i. . . « • 

attribute 240ofkeyobject230. Data indicating the keyboard p /°^. lfi0 of^ ^gmg mechanism provided by 

manipulation type "Hold" is stored in the keyboard manipu- the W * d ™ s95 °P eratin f system. The message includes the 

lation type attribute 242 of key object 230. Data indicating COmmand StnDg ******* ™ ih the voloe command, 

a repeat count of "1" is stored in repeat count attribute 244. Nole lhal while toe present invention has been described 

Data indicating a duration of three seconds is stored in 55 with reference to one mechanism for transmitting voice 

duration attribute 246. No text is stored in text attribute 248. command messages, other mechanisms are possible. 

Command set objects can contain any number of com- ^fore, it is understood that the present invention is not 

mand objects. Therefore, steps 420 through 480, during hmited t0 an y particular mechanism for transmitting mes- 

which data is collected for a command object, may be sages. 

repeated any number of times to populate a command set 60 rniuMANrn Mnnirr f 
object with any number of command objects. Similarly, <~ummajnd muuull 
command objects may contain any number of key objects. The command process 160 is the process that is respon- 
Therefore, steps 430 through 480, during which data is sible for receiving command strings that represent voice 
collected for a key object, may be repeated any number of commands from the sound engine 150. The command pro- 
times to populate a single command object 65 cess 160 is also responsible for determining whether the 
In this example, a command set object with a command command string received from the sound engine 150 is 
set name attribute value of "Fighter Jet" is created. A associated with any key combinations, and is responsible for 
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transmitting information about how to mimic the keyboard 
combination to the key mimicking mechanism 164. The 
keyboard mimicking mechanism is responsible for sending 
the keyboard combination as mimicked keyboard signal to 
the computer system 100. 

The command process 160 is started by the user. The 
techniques available for starting processes vary according to 
the operating system running on computer system 100. 
Because the techniques are well known, they shall not be 
further described. 

FIG. 5 is a flowchart depicting steps performed by com- 
mand process 160. To facilitate the description of the steps, 
an example is provided based on table 310, shown in FIG. 
3. Table 310 illustrates the values of attributes of three 
command objects that belong to the "Fighter Jet" command 
set object 210. Each entry in table 310 represents a key 
object contained in one of the command objects of the 
command set object 210. 

The value shown in the command string column for any 
particular entry in table 310 represents the command string 
attribute value of the command object that contains the key 
object represented by the entry. For each entry, the values 
shown in the key combination, keyboard manipulation type, 
the repeat count, and the duration columns represent the 
values of attributes for the particular object represented by 
the entry. For example, the entry 322 represents an key 
object contained by a command object with a command 
string attribute of 'Flare*. The key object has key 
combination, keyboard manipulation type, the repeat count 
column, and the duration attribute values of "Ctrl+R\ 'Hit', 
five (repeat count), zero (duration). 

For purposes of illustration, none of key objects repre- 
sented by table 310 have text attributes containing any value. 
Thus table 310 contains no column for text. 

At step 510, the command set is selected. An interface for 
selection of the file containing the command set object of the 
command set desired by the user is presented to the user. 
When the user selects a file, data is received indicating 
which file was selected. There are many well known mecha- 
nisms that may be used to provide an interface for selection 
of files, and therefore they shall not be further described. In 
this example, assume the user selects the file used to store 
the command set object with a command set name attribute 
value of "Fighter Jet". 

At step 520, an association between command strings and 
key combinations is established. For example, in an embodi- 
ment that uses the data structures shown in FIG. 2, the 
command set object 210, and the objects it contains (i.e. 
command objects and the key objects contained in the 
command objects) are retrieved from the file selected at step 
520. A list of the command objects in the command set 
object is generated. As previously mentioned, attributes of 
the key objects are considered to be mapped to the command 
string attribute of the command object that contains the key 
object. Thus the list of command objects embodies an 
association between command strings and key combina- 
tions. In this example, Table 310 illustrates the association 
between the command strings and key combinations. Each 
entry represents a mapping between the attributes of the 
object key represented by the entry and the command string 
shown for the entry. 

Steps 530 through 560 represent a loop for receiving 
command strings and locating the key combinations asso- 
ciated with the command strings. If a received command 
string is associated with at least one key combination, a 
message is transmitted to the keyboard mimicking mecha- 



20 



25 



nism. The keyboard mimicking mechanism shall be 
described in further detail. 

At step 530, a command string is received. The sound 
engine 150 transmits a message to the command process 160 
5 when the sound engine 150 receives a voice command. The 
message transmitted from the sound engine 150 includes the 
command string associated with the voice command. In this 
example, assume the sound engine is operating in full duplex 
mode and is generating sound when the user issues the voice 
10 command "air brakes". In response, the command string 
"AirBrakes" is received. 

At step 540, the key combination associated with the 
command suing, if any, is found. The list of objects is 
searched for a command object with a matching command 
15 string attribute. In this example, the command object with 
the command string attribute value of "AirBrakes" is found. 
The command object that is found is referred to herein as the 
requested command object. 

At step 550, it is determined whether at least one key 
combination associated with a command string was found in 
step 540. If a command object is found in step 550, then at 
least one key combination associated with the command 
string is found. If at least one key combination associated 
with a command string was found, then control passes to 
step 560. Otherwise execution ceases until another voice 
command is issued. In this example, because a command 
object was found in step 540, control passes to step 560. 
At step 560, key object messages are transmitted to the 
3Q keyboard mimicking mechanism. Each key object in the 
requested command object is processed. For each key 
object, a key object message containing data representing 
the attributes of the key object is transmitted to the keyboard 
mimicking mechanism. In transmitting the message for the 
35 key object to the keyboard mimicking mechanism 164, the 
key combination represented by the key object is placed in 
a queue by the keyboard mimicking mechanism 164. 

After step 560, execution ceases until another voice 
command is received. In this example, the command process 
40 160 iterates through the key object represented by entry 318 
(the only key object contained in the requested command 
object). The command process 160 sends to keyboard mim- 
icking mechanism 164 data representing the attributes of the 
key object represented by entry 318. Execution then ceases 
45 until subsequent voice command is received. 

KEYBOARD MIMICKING MECHANISM 

FIG. 6 is a flowchart showing steps performed by key- 
board mimicking mechanism 164. The act of receiving key 

50 combinations and generating signals that, to computer sys- 
tem 100, appear to be signals generated by a user pressing 
keys on the keyboard, is referred to as mimicking signals 
generated by a keyboard. Another process running on the 
computer system 100, such a computer game process, 

5S responds to these mimicked keyboard signals as it does to 
any signal actually generated by the keyboard. To facilitate 
the description of the steps, the example previously dis- 
cussed is continued. 

The keyboard mimicking mechanism 164 is shown in 

60 FIG. IB as a thread executing as part of command process 
160. Alternatively, the keyboard mimicking mechanism 164 
may be a separate process executing in an address space 
separate from the address space in which command process 
160 is executing. 

65 The keyboard mimicking mechanism 164 receives the key 
object messages from the command process 160. The data 
received in the key object message is added as an entry in a 
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FIFO queue. Each entry of the FIFO queue is an object sequences is repeatedly sent to the keyboard controller 158 
belonging to the object key class. An object that is an entry for the number of times corresponding to the repeat count, 
in the FIFO queue is referred to as a queued object. Because If the current queued object has a keyboard manipulation 
a queued object belongs to the key object class, each entry type attribute value of "hold", then the keyboard mimicking 
has a key combination, keyboard manipulation type, repeat 5 mechanism 164 sends a sequence of key-down instruction 
count, duration, and text attribute. sequences representing the particular key combination rep- 
Referring to FIG. 6, at step 610, the keyboard mimicking rcsented b * _J he curreDt 9 ueued t0 me keyboard 
mechanism 164 reads the next entry from the FIFO queue. C0D . 0 er * 

If there is no entry in the queue, the execution of steps 610 V™^- rf the J t f 1 attribute contains ™V text > a pajr 

through 650 is suspended for a period time, or suspended » of key-down and key-up tnstrucuon sequences are sent for 

until a key object message is received. Tlie queued key ea< * ch , a ^ ct " m * e te * m order * e . charact . er »n»P 

• • i • « • £ i /\ • c j \. \ m the text. The pair of key-down and key-up instruction 

object that is read in step 610 is referred to as the current ^ ^ fo P eacfa ^ for £ £ 

queued object. In this example assume there is one entry in ^ ^ ^ tQ ^ charactef m / n {Q 

the queue, which is the queued object represented by entry step 

318 in Table 310. 15 ^ ^ e p resen t example, the current queued object's key- 

At step 620, the key combination represented by the board manipulation type attribute value is "hold". Key-down 

current queued key object is sent to the computer system 100 instruction sequences representing the key combination 

as mimicked signals from the keyboard. The manner in ' Ctrl +A' are sent to the keyboard controller 158. Specifically, 
which the signals are mimicked depends on the attributes of a key-down instruction sequence representing the depres- 

the current queued object. 20 sion of the 'Ctrl' key is sent, followed by a key-down 

According to an embodiment of the invention, keyboard instruction sequence representing the depression of the key 
controller 158 supports at least three controller commands: 

a new instruction command, a key command, and a send key At ste P 630 > il is determined whether the duration attribute 

command. The new instruction command indicates to the value of tne current queued object is greater than zero. If the 

keyboard controller 158 that a new instruction representing 25 duration is not greater than zero, then the queued object is 

a data byte is about to be sent. The new instruction command removed from the queue and control passes to step 610. 

is transmitted by writing a command byte to a first port Otherwise, control passes to step 640. In this example the 

address configured to accept the command byte. dur A atl0n ,s *™ f c ° nds > 50 f° n1 f°\ f asses t0 ? e P ™- „ 

| j , j i .i .1 i At step 640, the keyboard mimicking mechanism 164 

^e key command represents a key, and whether the key 3Q pause5 ex ^ cu u 0 n of me steps for me period of time specified 

is being depressed or released. The key command is trans- b me dujation aUribute Tne k ^ mimicking ^ echa _ 

mined by wntmg a data byte to a second port address nism 164 ^ a Umer mec hanism to wait for the period of 

configured to accept the data byte. time specified. The use of timer mechanisms are well know 

The send key command signals the keyboard controller to to those skilled in the art and shall not be further described, 

send the key represented by the key command to the 35 After the period of time has expired, control passes to step 

computer system 100 as a signal generated by the keyboard. 650. The effect of pausing in this manner is to signal to the 

This command is transmitted by writing a command byte to computer system 100 that the key combination is being 

the first port address configured to accept such a command depressed for the period of lime corresponding to the 

byte. duration attribute value. When the key combination is being 

Transmitting a sequence of controller commands to the 40 signaled to the computer system 100 in this manner, the key 

keyboard controller 158 will cause the keyboard controller combination is considered "held". In this example, the 

to send signals to the computer system 100 that indicate that keyboard mimicking mechanism 164 waits for a period of 

a particular key has been depressed or has been released. A three seconds. Then control passes to step 650. 

sequence of controller commands that cause the keyboard In step 650, a key combination is sent to the computer 

controller 158 to send a signal that a particular key combi- 45 system 100 as a mimicked signal from the keyboard to 

nation is being depressed is referred to as a key-down "release" the key combination being held. In this example, 

instruction sequence. A sequence of controller commands the key-up instruction sequence for the key 'A' is sent to the 

that causes the keyboard controller 158 to send a signal that keyboard controller 158, followed by a key-up instruction 

a particular key combination is being released is referred to sequence for the key 'Ctrl'. 

as a key-up instruction sequence. 50 l n step 660, it is determined whether the "holding" and 
If the current queued object has a keyboard manipulation "releasing" of the key combination represented by the 
type attribute value of "hit", the key mimicking mechanism current queued object has been repeated the number of times 
sends a sequence of key-down instruction sequences and specified in the repeat count. If is determined that "holding" 
key-up instruction sequences representing the particular key and "releasing" of the key combination represented by the 
combination represented by the current queued object to the 55 current queued object has been repeated the number of times 
keyboard controller. If the repeat count for the queued object specified in the repeat count, then control passes to step 610. 
is greater than one, the sequence of key-down instruction Otherwise, control passes to step 620. 
sequences and key-up instruction sequences is repeatedly Note that when the current queued object has a keyboard 
sent to the keyboard controller 158 for the number of times manipulation type attribute value of 'hold' and a duration 
corresponding to the repeat count. 60 attribute value of zero, no corresponding key-up instruction 
If the current queued object has a keyboard manipulation sequence is sent at step 650. The key combination repre- 
type attribute value of "release", then the keyboard mim- sented by the queued object is to be held indefinitely. The 
icking mechanism 164 sends a sequence of key-up ins true- key combination is released only when the keyboard mira- 
tion sequences representing the particular key combination icking mechanism 164 performs the step 630 on a queued 
represented by the current queued object to the keyboard 65 object having the same key combination attribute value and 
controller 158. If the repeat count for the queued object is a keyboard manipulation type attribute value of 'release' (or 
greater than one, the sequence of key-up instruction 'hit'). 
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Assume that the current queued object in the present After the command objects and key objects are stored in 

example has a duration attribute value representing an a command file, the command process 160 is started. The 

indefinite duration. Then at step 630, it is determined that command file containing the command set to be used with 

duration attribute value of current queued object has a the particular application is selected. Then the user uses the 

duration not greater than zero. Therefore, control passes to 5 voice recognizer software to make the desired set of voice 

step 620, leaving the key combination represented by the commands the active set of voice commands. Finally, the 

current queued objects as a key combination being held user speaks the voice commands to control the particular 

indefinitely. application. The particular application may be controlled by 

Assume that after a few seconds the next key object voice commands without ever being configured to interface 

message is received, and he data from the message is 10 with any particular sound engine. 

inserted into the queue. The queued object just inserted is In an alternate embodiment of the invention, a repeat 

represented by the entry 320, and because it is the next count ("command level repeat count") is associated with 

queued object to be read at step 610, that queued object is each command object. Each command object has a com- 

the current queued object. The queued object corresponds to mand level repeat count attribute. The command level repeat 

the key-combination that is mapped to the command string 15 count represents the number of times to repeat the series of 

'ReleaseBrakes*. keyboard commands represented by the set of key objects 

At step 630, in the present example, a key-up instruction associated with the requested command object, 

sequence is sent for the key 'A', followed by a key- up For example, assume that in a fighter jet simulator it 

instruction sequence for the key 'Ctrl'. Sending the key-up desirable to release flares, and to immediately follow the 

instruction sequences in this manner releases the key com- 20 release of flares with the release of chaff, and to repeat this 

bination that was being held indefinitely. sequence of releasing flares and chaff three times. Chaff 

Assume now that the current queued object is represented comprises strands of metal which are expelled from an 

by entry 322 in FIG. 3. At step 620, the keyboard mimicking airplane in order to confuse missiles guided by radar. A 

mechanism 164 transmits a key-down instruction sequence command object associated with the voice command "flaff", 

for 'Ctrl', followed by a key-down instruction sequence for 25 and associated with key objects representing the keyboard 

'R\ This pair of key-down instruction sequences is imme- commands necessary to release the flares and chaff, is 

diately followed by a key-up instruction sequence for 'R\ created in manner described previously, 

and then by a key-up instruction sequence for 'Ctrl'. In addition, steps for collecting data to create a command 

Because the repeat count attribute value of the current 3Q object (shown in FIG. 4) include a step to receive data 

queued object is five, the two pairs of key-down and key-up representing the command level repeat count attribute for 

instruction sequences are repeatedly sent five times before the command object. Assume in this example, data repre- 

execution of the steps proceeds to step 630. senting a command level repeat count value of three is 

While one keyboard mimicking mechanism 164 has been received. Then data needed to create the key objects repre- 

described, others alternatives are possible. Therefore, it is 35 senting the release flare keyboard command ('Ctrl+R') and 

understood that the present invention is not limited to any release chaff keyboard command ('Ctrl+H') is received, 

particular mechanism for mimicking key combinations as When the voice command "flaff" is issued by a user, then 

signals from a keyboard. For example, one alternate key- at step 560 (FIG. 5), the step for transmitting a key object 

board mimicking mechanism 164, operating on a computer message for each key object associated with the requested 

system 100 running on the Windows95 operating system, ^ command object, is repeated for the number of times indi- 

may send a key-down and key-up event messages to the cated by the repeat count attribute of the requested command 

Windows95 operating system. The computer system 100 object, which is three. Thus, the set of key objects repre- 

responds to these event messages as signals generated by a sen ted by requested command object is placed in the FIFO 

keyboard. queue for the number of times indicated by the repeat count 

The key combinations sent by the keyboard mimicking 45 attribute. Thus a key object representing the release of flares 

mechanism 164 appear as regular keyboard commands to ^ placed in the queue, followed by the key object repre- 

software running on computer system 100. Many software senting the release of chaff. Then another pair of these key 

applications, such as computer games, are configured to objects is placed in the queue, followed by a third pair, 

respond to such keyboard commands. This feature allows The keyboard mimicking mechanism operates in the 

the user who desires to control a software application 50 manner similar to that described earlier. The keyboard 

through voice commands, to select a particular sound engine mimicking mechanism processes the key objects in the order 

from the many sound engines available on the market, even in which they were placed in the queue. As a result, key 

though the particular sound engine is not specifically con- combinations representing three pairs of a release of flares 

figured to interface with the software application. and a subsequent release of chaff, are transmitted as mim- 

To use a desired set of voice commands to control a 55 icked keyboard signals, 

particular application not configured to interface with the In the foregoing specification, the invention has been 

sound engine, the user first creates, using the voice recog- described with reference to specific embodiments thereof. It 

nizer software, a set of desired voice commands. Each of the will, however, be evident that various modifications and 

voice commands are mapped to a command string. Then the changes may be made thereto without departing from the 

user creates a file representing a command set containing the 60 broader spirit and scope of the invention. The specification 

commands to be used in conjunction with the particular and drawings are, accordingly, to be regarded in an illus- 

application. Each of the commands are associated with a trative rather than a restrictive sense, 

command string mapped to a voice command by creating What is claimed is: 

command objects with matching command string attributes 1- A method for sending signals to a computer system, the 

in the manner described. For each of the command objects, 65 method comprising the steps of: 

key objects corresponding to the desired command set establishing an association between a set of command 

commands are generated. strings and a set of keyboard combinations, wherein 
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each command string from said set of command strings 
is associated with at least one keyboard combination 
from said set of keyboard combinations; 

receiving from a sound engine a command string gener- 
ated in response to a voice command; 5 

selecting a keyboard combination based on said associa- 
tion between the set of command strings and the set of 
keyboard combinations; and 

sending the keyboard combination to the computer system 1Q 
to mimic signals generated by a keyboard; and 

generating sound output from the sound engine while said 
sound engine is receiving said voice command and 
operating in full duplex mode. 

2. The method of claim 1, wherein the step of establishing ^ 
the association between the set of command strings and the 
set of keyboard combinations includes establishing an asso- 
ciation between the set of command strings and a set of 
keyboard combinations that control play in a computer 
game. 20 

3. The method of claim 1, wherein the step of sending the 
keyboard combination includes the steps of: 

sending the keyboard combination to a keyboard control- 
ler; and 

sending a set of messages to the keyboard controller to 25 
send the key combination to the computer system as a 
signal generated from the keyboard. 

4. The method of claim 1, wherein the step of sending the 
keyboard combination includes the step of sending a key- 
board message to an operating system running on said 30 
computer system, the keyboard message being a message to 
which said operating system responds as signals generated 
by said keyboard. 

5. A method for sending signals to a computer system the 
method comprising the steps of: 35 

establishing an association between a set of command 
strings and a set of keyboard combinations, wherein 
each command string from said set of command strings 
is associated with at least one keyboard combination 
from said set of keyboard combinations; 40 

receiving from a sound engine a command string gener- 
ated in response to a voice command; 

selecting a keyboard combination based on said associa- 
tion between the set of command strings and the set of 45 
keyboard combinations; and 

sending the keyboard combination to the computer system 
to mimic signals generated by a keyboard; 

receiving from a user the command string; 

receiving from a user the keyboard combination; 50 

receiving from a user data identifying the type of key- 
board manipulation; and 

storing a mapping of said command string to said key- 
board combination and the type of keyboard manipu- 55 
lation; 

wherein the step of receiving from a user data identifying 
the type of keyboard manipulation includes the steps 
of: 

receiving data representing a duration associated with go 
the type of keyboard manipulation if the data iden- 
tifying the type of keyboard manipulation identifies 
the type of keyboard manipulation as a bold type; 

storing a mapping of the command string to the dura- 
tion; 65 
wherein the step of sending the keyboard combination 

further includes, if the duration associated with the 
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keyboard combination is set to a value that represents 
infinity, the step of sending the selected keyboard 
combination to the computer system to mimic signals 
generated by the keyboard as a keyboard combination 
that is depressed indefinitely; 
wherein the method further comprises the steps of 
receiving a subsequent command string from the sound 
engine, said subsequent command string being asso- 
ciated with the keyboard combination and a release 
type of keyboard manipulation; and 
in response to receiving said subsequent command 
string, sending to the computer system a keyboard 
combination to mimic signals generated by the key- 
board that signal to said computer system that the 
keyboard combination that is depressed indefinitely 
is no longer depressed. 

6. A computer-readable medium carrying sequences of 
instructions for sending signals to a computer system, the 
sequences of instructions including instructions which, 
when executed by one or more processors, cause the one or 
more processors to perform the steps of: 

establishing an association between a set of command 
strings and a set of keyboard combinations, wherein 
each command string from said set of command strings 
is associated with at least one keyboard combination 
from said set of keyboard combinations; 

receiving from a sound engine a command string gener- 
ated in response to a voice command; 

selecting a keyboard combination based on said associa- 
tion between the set of command strings and the set of 
keyboard combinations; and 

sending the keyboard combination to the computer system 
to mimic signals generated by a keyboard; and 

generating sound output from the sound engine while said 
sound engine is receiving said voice command and 
operating in full duplex mode. 

7. The computer-readable medium of claim 6, wherein the 
step of establishing the association between the set of 
command strings and the set of keyboard combinations 
includes establishing an association between the set of 
command strings and a set of keyboard combinations that 
control play in a computer game. 

8. The computer-readable medium of claim 7, wherein the 
step of sending the keyboard combination includes the steps 
of: 

sending the keyboard combination to a keyboard control- 
ler; and 

sending a set of messages to the keyboard controller to 
send the key combination to the computer system as a 
signal generated from the keyboard. 

9. The computer-readable medium of claim 7, wherein the 
step of sending the keyboard combination includes the step 
of sending a keyboard message to an operating system 
running on said computer system, the keyboard message 
being a message to which said operating system responds as 
signals generated by said keyboard. 

10. A computer-readable medium carrying sequences of 
instructions for sending signals to a computer system, the 
sequences of instructions including instructions which, 
when executed by one or more processors, cause the one or 
more processors to perform the steps of: 

establishing an association between a set of command 
strings and a set of keyboard combinations, wherein 
each command string from said set of command strings 
is associated with at least one keyboard combination 
from said set of keyboard combinations; 
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receiving from a sound engine a command string gener- 
ated in response to a voice command; 

selecting a keyboard combination based on said associa- 
tion between the set of command strings and the set of 
keyboard combinations; and 5 

sending the keyboard combination to the computer system 
to mimic signals generated by a keyboard; 

receiving from a user the command string; 

receiving from a user the keyboard combination; jq 

receiving from a user data identifying the type of key- 
board manipulation; and 

storing a mapping of said command string to said key- 
board combination and the type of keyboard manipu- 
lation; 15 

wherein the step of receiving from a user data identifying 
the type of keyboard manipulation includes the steps 
of: 

receiving data representing a duration associated with 
the type of keyboard manipulation if the data iden- 20 
tifying the type of keyboard manipulation identifies 
the type of keyboard manipulation as a hold type; 
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storing a mapping of the command string to the dura- 
tion; 

wherein the step of sending the keyboard combination 
further includes, if the duration associated with the 
keyboard combination is set to a value that represents 
infinity, the step of sending the selected keyboard 
combination to the computer system to mimic signals 
generated by the keyboard as a keyboard combination 
that is depressed indefinitely; 

wherein the computer-readable medium carries 
sequences of instructions for receiving a subsequent 
command string from the sound engine, said subse- 
quent command string being associated with the 
keyboard combination and a release type of key- 
board manipulation; and 

in response to receiving said subsequent command 
sting, sending to the computer system a keyboard 
combination to mimic signals generated by the key- 
board that signal to said computer system that the 
keyboard combination that is depressed indefinitely 
is no longer depressed. 

***** 
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